Temporal compressive sensing systems

ABSTRACT

Methods and systems for temporal compressive sensing are disclosed, where within each of one or more sensor array data acquisition periods, one or more sensor array measurement datasets comprising distinct linear combinations of time slice data are acquired, and where mathematical reconstruction allows for calculation of accurate representations of the individual time slice datasets.

CROSS-REFERENCE

This application claims the benefit of U.S. Provisional Application No.62/258,194, filed on Nov. 20, 2015, which application is incorporatedherein by reference.

STATEMENT AS TO FEDERALLY SPONSORED RESEARCH

This invention was made with the support of the United States governmentunder Award number DE-SC0013104 by the United States Department ofEnergy.

BACKGROUND

Compressive sensing is an approach to signal acquisition and processingthat makes use of the inherent properties of some signals to measure andmathematically reconstruct the signal based on a limited series of testmeasurements. This disclosure relates to novel systems and methods fortemporal compressive sensing. For example, one specific disclosure isrelated to novel temporal compressive sensing systems and methods asapplied to a transmission electron microscope (TEM).

SUMMARY

Disclosed herein are methods for temporal compressive sensing,comprising: a) directing radiation having an intensity from a sourcetowards a sample or scene; b) capturing sensor array data for one ormore data acquisition periods, wherein within each of the one or moredata acquisition periods, one or more measurement datasets correspondingto distinct linear combinations of patterns of the radiationtransmitted, reflected, elastically scattered, or inelasticallyscattered by the sample or scene are captured for a series of timeslices; and c) reconstructing a time slice dataset for each of the timeslices of the series within each of the one or more data acquisitionperiods using: i) the one or more measurement datasets captured for eachdata acquisition period; ii) a series of coefficients that describe aknown time-dependence of the intensity of the radiation from the sourcethat is directed to the sample or scene within the data acquisitionperiod, or a known time-dependence for switching the radiationtransmitted, reflected, elastically scattered, or inelasticallyscattered by the sample or scene to different regions of the sensorarray within the data acquisition period, wherein the coefficients varyas a function of time slice and region of the sensor array but areindependent of the spatial position for a given pixel within the sensorarray or within a given region of the sensor array; and iii) analgorithm that calculates the time slice datasets from the one or moremeasurement datasets captured for each data acquisition period and theseries of coefficients; thereby providing a series of time slicedatasets for each of the one or more data acquisition periods that has atime resolution exceeding the time resolution determined by the lengthof the data acquisition period.

In some embodiments, the sensor array is a two-dimensional sensor arraycomprising a charge-coupled device (CCD) sensor, a complementary metaloxide semiconductor (CMOS) sensor, a CMOS framing camera, a photodiodearray, or any combination thereof. In some embodiments, the sensor arrayfurther comprises a nonlinear optical material, a fluorescent material,a phosphorescent material, or a micro-channel plate, that converts theradiation into radiation directly detectable by the sensor array. Insome embodiments, the algorithm used to reconstruct the time slicedatasets is an optimization algorithm that penalizes non-sparsesolutions of an underdetermined system of linear equations via the 11norm, the total number of non-zero coefficients, total variation, orbeta process priors; an iterative greedy recovery algorithm; adictionary learning algorithm; a stochastic Bayesian algorithm; avariational Bayesian algorithm; or any combination thereof. In someembodiments, at least or at least about 10 time slice datasets arereconstructed from the one or more measurement datasets captured foreach data acquisition period. In some embodiments, the two-dimensionalsensor array operates at an effective data acquisition and read-out rateof at least or at least about 100 frames per second. In someembodiments, the radiation comprises electrons, and wherein the sensorarray is a charge-coupled device (CCD) sensor, an image-intensifiedcharge-coupled device (ICCD) sensor, the detector in an electron energyloss spectrometer (EELS), or any combination thereof. In someembodiments, the radiation comprises electrons and the sensor array isreplaced by the detector in an energy-dispersive x-ray spectrometer(EDX). In some embodiments, the time slice data sets comprisereconstructed frames of transmission electron microscope image data,transmission electron microscope diffraction pattern data, transmissionelectron microscope electron energy loss spectral data, transmissionelectron microscope energy-dispersive x-ray spectral data, or scanningelectron microscope image data. In some embodiments, the number of timeslice datasets to be reconstructed is adjusted during the calculation ofthe time slice datasets. In some embodiments, the number of time slicedatasets to be reconstructed is optimized by calculating a range ofmeasurement matrix coefficients, each with a different number of timeslices, prior to capturing the measurement datasets. In someembodiments, the distinct linear combinations of patterns of theradiation transmitted, reflected, elastically scattered, orinelastically scattered by the sample or scene for a series of timeslices are generated by modulating in a temporal fashion an experimentalparameter other than the radiation intensity. In some embodiments, theexperimental parameter to be temporally modulated is selected from thegroup consisting of rotational orientation of the sample or scene,linear translation of the sample or scene in one dimension, lineartranslation of the sample or scene in two dimensions, and lineartranslation of the sample or scene in three dimensions, or anycombination thereof. In some embodiments, the radiation is focused to anarrow beam and the experimental parameter to be temporally modulated isthe position of the beam relative to the sample or scene. In someembodiments, the series of coefficients describe a knownspatial-dependence and time-dependence of the intensity of the radiationfrom the source that is directed towards the sample or scene within thedata acquisition period, or a known spatial-dependence of the intensityof the radiation from the source and a known time-dependence forswitching the radiation transmitted, reflected, elastically scattered,or inelastically scattered by the sample or scene to different regionsof the sensor array within the data acquisition period.

Also disclosed herein are systems for temporal compressive sensing,comprising: a) a radiation source that provides radiation having anintensity directed towards a sample or scene; b) a sensor array thatdetects the radiation subsequent to transmission, reflection, elasticscattering, or inelastic scattering by the sample or scene; c) amechanism that rapidly modulates the intensity of the radiationgenerated by the radiation source prior to its interaction with thesample or scene, or that rapidly switches the radiation transmitted,reflected, elastically scattered, or inelastically scattered by thesample or scene to different regions of the sensor array, and d) one ormore computer processors that: (i) capture sensor array data for one ormore data acquisition periods, wherein within each data acquisitionperiod, one or more measurement datasets corresponding to distinctlinear combinations of patterns of transmitted, reflected, elasticallyscattered, or inelastically scattered radiation for a series of timeslices are captured; and (ii) reconstruct a time slice dataset for eachtime slice within each of the one or more data acquisition periodsusing: 1) the one or more measurement datasets captured for each dataacquisition period; 2) a series of coefficients that describe a knowntime-dependence of the intensity of the radiation generated by theradiation source and directed to the sample or scene within the dataacquisition period, or a known time-dependence for switching theradiation transmitted, reflected, elastically scattered, orinelastically scattered by the sample or scene to different regions ofthe sensor array within the data acquisition period, and wherein thecoefficients vary as a function of time slice and region of the sensorarray but are independent of the spatial position for a given pixelwithin the sensor array or within a given region of the sensor array;and 3) an algorithm that calculates the time slice datasets from the oneor more measurement datasets captured for each data acquisition periodand the series of coefficients; thereby generating a series of timeslice datasets for each of the one or more data acquisition periods thathas a time resolution exceeding the time resolution determined by thelength of the data acquisition period.

In some embodiments, the radiation source is a laser, a photocathode, anelectron gun, or any combination thereof. In some embodiments, thesensor array is a two-dimensional sensor array comprising acharge-coupled device (CCD) sensor, a complementary metal oxidesemiconductor (CMOS) sensor, a CMOS framing camera, a photodiode array,or any combination thereof. In some embodiments, the sensor arrayfurther comprises a nonlinear optical material, a fluorescent material,a phosphorescent material, or a micro-channel plate, that converts thesignal from the radiation source of claim 1 into radiation directlydetectable by the sensor array. In some embodiments, the algorithm thatreconstructs the time slice datasets is an optimization algorithm thatpenalizes non-sparse solutions of an underdetermined system of linearequations via the 11 norm, the total number of non-zero coefficients,total variation, or beta process priors, an iterative greedy recoveryalgorithm, a dictionary learning algorithm, a stochastic Bayesianalgorithm, a variational Bayesian algorithm, or any combination thereof.In some embodiments, at least or at least about 10 time slice datasetsare reconstructed from the one or more measured datasets captured foreach data acquisition period. In some embodiments, the two-dimensionalsensor array operates at an effective data acquisition and read-out rateof at least or at least about 100 frames per second. In someembodiments, the radiation comprises electrons and the sensor array is acharge-coupled device (CCD) sensor, an image-intensified charge-coupleddevice (ICCD) sensor, the detector in and electron energy lossspectrometer (EELS), or any combination thereof. In some embodiments,the radiation comprises electrons and the sensor array is replaced bythe detector in an energy-dispersive x-ray spectrometer (EDX). In someembodiments, the time slice data sets comprise reconstructed frames oftransmission electron microscope image data, transmission electronmicroscope diffraction pattern data, transmission electron microscopeelectron energy loss spectral data, transmission electron microscopeenergy-dispersive x-ray spectral data, or scanning electron microscopeimage data. In some embodiments, the number of time slice datasets to bereconstructed is adjusted during the calculation of the time slicedatasets. In some embodiments, the number of time slice datasets to bereconstructed is optimized by calculating a range of measurement matrixcoefficients, each with a different number of time slices, prior tocapturing the measurement datasets. In some embodiments, the series ofcoefficients describe a known spatial-dependence and time-dependence ofthe intensity of the radiation from the source that is directed towardsthe sample or scene within the data acquisition period, or a knownspatial-dependence of the intensity of the radiation from the source anda known time-dependence for switching the radiation transmitted,reflected, elastically scattered, or inelastically scattered by thesample or scene to different regions of the sensor array within the dataacquisition period.

Disclosed herein are systems for temporal compressive sensing,comprising: a) a radiation source that provides radiation directedtowards a sample or scene; b) a sensor array that detects the radiationsubsequent to transmission, reflection, elastic scattering, or inelasticscattering by the sample or scene; c) a mechanism that rapidly modulatesthe one-, two-, or three-dimensional translational position orrotational orientation of the sample or scene relative to the directionof irradiation; and d) one or more computer processors that: (i) capturesensor array data for one or more data acquisition periods, whereinwithin each data acquisition period, one or more measurement datasetscorresponding to distinct linear combinations of patterns oftransmitted, reflected, elastically scattered, or inelasticallyscattered radiation for a series of time slices are captured; and (ii)reconstruct a time slice dataset for each time slice within each of theone or more data acquisition periods using: 1) the one or moremeasurement datasets captured for each data acquisition period; 2) aseries of coefficients that describe a known time-dependence of thetranslational position or rotational orientation of the sample or scenewithin the data acquisition period; and 3) an algorithm that calculatesthe time slice datasets from the one or more measurement datasetscaptured for each data acquisition period and the series ofcoefficients; thereby generating a series of time slice datasets foreach of the one or more data acquisition periods that has a timeresolution exceeding the time resolution determined by the length of thedata acquisition period.

In some embodiments, the radiation is focused to a narrow beam and themechanism rapidly modulates the position of the beam relative to thesample or scene.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference in their entirety tothe same extent as if each individual publication, patent, or patentapplication was specifically and individually indicated to beincorporated by reference in their entirety. In the event of a conflictbetween a term herein and a term in an incorporated reference, the termherein controls.

BRIEF DESCRIPTION OF THE DRAWINGS

The following description and examples illustrate embodiments of theinvention in detail. It is to be understood that this invention is notlimited to the particular embodiments described herein and as such mayvary. Those of skill in the art will recognize that there are numerousvariations and modifications of this invention, which are encompassedwithin its scope.

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1 illustrates 10 frames of TEM image data from an in situ tensilecrack propagation experiment (courtesy K. Hattar et al., Sandia NationalLaboratory).

FIG. 2 illustrates different combinations of ten time slice datasetsthat are sent to four different regions of a large camera using a fastswitching system, and digitally segmented into four image frames (i.e.,a 2×2 array of images captured by the large camera sensor) for analysis.The mask matrix, also called the measurement matrix, is in this case a4×10 array of real numbers specifying the coefficients expressing eachof the four measured frames as a linear combination of the image datafrom ten distinct time slices.

FIG. 3 illustrates four segmented image frames captured during a singlecamera data acquisition period using the different combinations of tentime slice datasets illustrated in FIG. 2 and a fast switching system.The four segmented image frames are captured simultaneously during asingle camera data acquisition period.

FIG. 4 illustrates the ten time slice images (datasets) reconstructedfrom the four segmented images illustrated in FIG. 3. The agreementbetween FIG. 1 and FIG. 4 illustrates that the data in FIG. 1 iscompressible, requiring only four measured images to reconstruct all tendistinct images representing the state of the sample in each time slice.

FIG. 5 depicts a generic, simplified schematic of the basic componentsand function of a TEM.

FIG. 6 illustrates one non-limiting example of a modified TEM thatutilizes a high-speed deflector system to implement the compressivesensing methods disclosed herein.

FIG. 7 illustrates one non-limiting example of a stroboscopic,time-resolved TEM that utilizes an arbitrary-waveform laser (e.g., withsub-picosecond-scale modulation and sub-nanosecond-scale pulse duration,or with nanosecond-scale modulation and microsecond-scale pulseduration) to modulate the current from a photoelectron source.

FIG. 8 illustrates one non-limiting example of an optical system(simplified schematic) for implementing the temporal compressive sensingmethods disclosed herein.

FIG. 9 illustrates one example of a computer system that may be used forimplementing the temporal compressive sensing data acquisition andanalysis methods of the present disclosure.

DETAILED DESCRIPTION

Overview of compressive sensing: Compressive sensing (also known ascompressed sensing, compressive sampling, or sparse sampling) is afamily of signal acquisition and processing techniques for efficientlyacquiring and reconstructing a signal. As used herein, the term “signal”and its grammatical equivalents includes, but is not limited to,intensity, frequency, or phase data as it pertains to an electrical,electromagnetic, or magnetic field, as well as to optical or non-opticalimage data, spectral data, diffraction data, and the like. Incompressive sensing, reconstruction of a signal is performed by making alimited number of signal measurements according to a defined set ofsampling functions (or test functions), and subsequently findingmathematical solutions to the resulting system of linear equations thatrelate the unknown “true” signal to the set of measured values.Reconstruction thus provides an estimate of the “true” signal, theaccuracy of which is dependent on several factors including, but notlimited to, properties of the signal itself, the choice of testfunctions used to sample the signal, the amount of noise in the signal,and the mathematical algorithm selected to solve the system of linearequations. Because the signal is under-sampled, the system of linearequations is underdetermined (i.e., has more unknowns than equations).In general, underdetermined systems of equations have an infinite numberof solutions. The compressive sensing approach is based on the principlethat prior knowledge of or reasonable assumptions about the propertiesof the signal can be exploited to recover it from far fewer samplingmeasurements than would be required by conventional Nyquist-Shannonsampling. Two conditions must be satisfied for accurate reconstructionof compressively sensed signals: (i) the signal must be “sparse” in somedomain (i.e., the signal may be represented in some N-dimensionalcoordinate system as a linear combination of basis vectors, where only asmall number, K, of the coefficients for each of the basis vectors arenon-zero (K<<N)), and (ii) the signal and sampling measurement functionsmust be incoherent (i.e., the set of measurement functions (vectors) arerandomly distributed across the set of N basis vectors for the domain inwhich the signal is sparse).

Many real world signals, e.g., photographic images and video data,exhibit underlying structure and redundancy that satisfy the sparsityand incoherence conditions in an appropriately selected domain. Datacompression and decompression algorithms used to produce mpeg and jpegfiles exploit essentially the same concept as that used in compressivesensing to reduce the amount of data storage required or to facilitatedata transmission. However, these signal processing algorithms areapplied post-signal acquisition. Compressive sensing is applied at thesignal acquisition stage to improve the efficiency of data capture aswell as to reduce data storage and transmission requirements.

In compressive sensing, a system of linear equations is generatedthrough acquisition of a series of sampling measurements performed usinga set of known test functions, where the total number of samplingmeasurements, M, is small compared to the number required byNyquist-Shannon sampling theory but where the sampled data stillcontains essentially all useful information contained in the originalsignal. This linear system of equations is often expressed as:y(m)=Φx(n)=ΦΨα  (1)where y(m), m=1, 2, . . . , M represents the sampling measurements,x(n), n=1, 2,. . . , N represents the values of the unknown signal, Φ isan M×N matrix representing the known weighting factors (test functions)used to acquire the sampling measurements (the latter comprising linearcombinations of the products of the weighting factors and the signalcoefficients for the chosen set of basis vectors), and Ψ and α representthe basis vectors and corresponding coefficients respectively of theN-dimensional coordinate system in which the signal, x(n), may berepresented as x(n)=Σ_(i=a) ^(N)α_(i)Ψ_(i). Solving equation (1) for theunknown values of x(n) thus corresponds to solving the underdeterminedsystem of linear equations. As indicated above, underdetermined systemsof linear equations have an infinite number of solutions, however,imposing the constraints of sparsity and incoherence limits the possiblesolutions to those having a small (or minimum) number of non-zerocoefficients, and enables one to reconstruct the original signal with ahigh degree of accuracy. A variety of mathematical approaches exist forsolving this problem including, but not limited to, optimization of thel₁ norm, greedy algorithms, stochastic Bayesian algorithms, variationalBayesian algorithms, and dictionary learning algorithms.

Video compressive sensing: The compressive sensing literature includesapplication areas ranging from optical imaging to magnetic resonanceimaging to spectroscopy and others. Temporal compressive sensingmethods, i.e., in which signals are reconstructed using data sets thatunder-sample the signal in the time domain, have been applied primarily,but not exclusively, to video compression. Typically these methodsutilize some form of a jittered, random-coded aperture that isphysically moved (usually with a piezoelectric system) on a time scalemuch shorter than the acquisition time for a single video frame, therebyspatially encoding the sampling measurements. Thus, in effect, the datumfor each pixel in the acquired video frame represents a different linearcombination of light intensities sampled at different points in time.Mathematical reconstruction is used to calculate the video image thatwould have been observed at each of the referenced points in time if theframe rate or data acquisition time for the camera had been faster. Infavorable cases, variants of the standard algorithms described in thecompressive sensing literature can be used to reconstruct tens or evenhundreds of reconstructed frames of video data from a single such dataacquisition period. This type of compressive sensing system has beendemonstrated for optical video cameras, and researchers are currentlyattempting to apply the same approach to compressive sensing intransmission electron microscopes (TEMs).

Video compressive sensing as applied to electron microscopy: Thedifficulties of producing the required coded aperture, inserting it atan appropriate place in the electron beam path, preventing it fromaccumulating contamination or being damaged upon exposure to theelectron beam, and moving it inside the vacuum system with the requiredspeed, precision, and repeatability have reportedly been substantial(see the recently published paper by Stevens et al., (2015), “ApplyingCompressive Sensing to TEM Video: a Substantial Frame Rate Increase onany Camera”, Adv. Structural and Chemical Imaging 1:10, for adescription of the computational and mathematical aspects of theapproach). The practical limitations of implementing coded-aperturevideo compressive sensing in a TEM have been and will continue to besubstantial. The system modifications required to implementcoded-aperture video compression can be both expensive and highlyinvasive, and may require frequent (and potentially difficult)maintenance and recalibration steps. The practicality of this approachwill thus likely be limited by physical considerations (charging,contamination, limited resolution, etc.) not accounted for in thepublished computational study.

U.S. Pat. No. 8,933,401 describes an alternative implementation ofcompressive sensing in an electron microscope system (including either aTEM or a scanning electron microscope (SEM)) in which a spatial patternof electron-illumination intensity (or “mask”) is produced at a sample,and the microscope captures information (including, but not limited to,image intensity data, diffraction patterns, electron energy-loss spectra(EELS), or energy-dispersive X-ray spectra (EDX)) using atwo-dimensional sensor array comprising N spatial pixels from thesuperposition of measurements at spatial positions defined by the mask.Rather than using a coded aperture to control the spatial variation ofelectron-illumination intensity, this approach makes use of an electronbeam scanning system configured to generate a plurality of electron beamscans over substantially an entire sample, with each scan varying inelectron-illumination intensity over the course of the scan. A set ofsampling measurements, captured using a number, M, of such spatialelectron-illumination intensity masks (where M<N) is used to reconstructthe image (or diffraction pattern, EELS, or EDX, etc.) that would havebeen produced had the measurement encompassed collecting data over theentire array of N spatial pixels for the full duration of the dataacquisition period. As mentioned above, any of a number of mathematicalreconstruction techniques can be used to solve the underdeterminedsystem of linear equations arising from the set of sampling measurementsto produce an accurate reconstruction of the original, full resolutionimage. Under favorable circumstances, such a system can be expected toacquire essentially the same information as a conventional TEM or SEMsystem, but with potentially much faster data acquisition times and muchsmaller data storage and handling requirements. The method was intendedprimarily for use in spatially-resolved diffraction and spectroscopymeasurements performed in a TEM, but the potential application space ismuch larger than this.

Time domain-encoded temporal compressive sensing: Disclosed herein is analternative approach to the temporal compressive sensing methoddescribed above (i.e., temporal compressive sensing in which the testfunctions are encoded in the time domain as opposed to the spatialdomain) that is potentially applicable to a wide variety of signalacquisition and processing fields in addition to optical video andelectron microscopy. In addition, several distinct hardwareimplementations of the approach are disclosed that enable operation invery different time domains (e.g., ranging from microsecond-scale topicosecond-scale time resolution).

To describe the new approach and distinguish it from previous work, westart by describing the existing approach of coded-aperture videocompressive sensing (i.e., spatially-encoded video compressive sensing)in more detail. In very general terms, coded-aperture video compressivesensing works by spatially-encoding multiple reconstructible frames ofvideo data into a single acquired video frame. We will describe anexample using typical values for operational parameters, with theunderstanding that the actual range of operational parameters inpractice can be quite large. An acquired video frame may, for example,be a single frame acquired by a charge-coupled device (CCD) cameraoperating in continuous acquisition mode at 100 Hz, so that each framerepresents an acquisition time of somewhat less than 10 milliseconds(after accounting for data read-out overhead). Throughout, we will referto this 10-millisecond span, which is the exposure time of aconventional acquisition system such as a camera, as a “block of time”.Thus, with a standard video acquisition system, one acquires one andonly one frame per block of time.

Now consider how the coded-aperture video compression system works.Suppose that the CCD camera has a 1024×1024 array of pixels. At anygiven instant within a 10 ms block of time, a coded aperture blocks orattenuates the signal reaching some fraction of the CCD pixels. Thiscoded aperture is capable of being physically moved very rapidly in aknown trajectory, so that it can be moved to 100 or more significantlydistinct locations during the 10 ms exposure time. Conceptually, we canbreak up the 10 ms exposure time into 100 distinct “time slices”, eachof which is 0.1 ms long. The intent is to determine what image wasstriking the full set of 1024×1024 pixels in each one of those 100 timeslices, or in other words, to calculate 100 reconstructed frames fromthe single 1024×1024-pixel acquisition. This is possible for tworeasons. First, each pixel is recording the total intensity from acertain known linear combination of the 100 time slices, and thecoefficients governing this linear combination are different fordifferent pixels. Therefore each pixel represents information from adifferent subset (or, more generally, weighted average) of the timeslices, and this means that there is information in the acquired imagethat in some respect distinguishes the 100 time slices from one another.Second, real-world video data generally has a high degree of informationredundancy, so that the actual number of independent data pointsrequired to describe, for example, a 1024 pixel×1024 pixel×100 framevideo is much less than the ˜10⁸ value one might expect from a simplecount of space-time voxels. Depending on the speed and degree ofcomplexity of the motion in the video, and the amount of distortionacceptable for a given application, data compression ratios of 10:1 or100:1 or even greater may be possible. There are multiple publishedexamples of coded-aperture optical video compressive sensing thatachieve compression rates of 100:1 or more, with moderate yet acceptablelevels of distortion. This distortion is considered to be a small priceto pay for effectively multiplying the frame rate (i.e., the dataacquisition and read-out rate) of an inexpensive camera by a factor of100 or more (i.e., the effective data acquisition and read-out rate, andthus the time resolution, of the camera exceeds that determined by itshardware limits).

This example illustrates the reconstruction of 100 “time slice” videoframes of 0.1 ms duration, each with 1024×1024 pixels, from a single 10ms acquired video frame of 1024×1024 pixels. Each pixel in the acquiredframe represents a different linear combination of information (asdetermined by the series of spatial masks used during acquisition) fromthe same spatial location in the 100 different time slices, and weacquire one frame per 10 ms block of time. In mathematical terms, thiscan be expressed as:M _(ij)=Σ_(k) c _(ijk) V _(ijk)+noise,  (2)where M_(ij) are the measured video frames (comprising the complete setof pixel data, such that indices i and j represent rows and columns inthe image, respectively), V_(ijk) are the video frames to bereconstructed (i.e., the set of N time slice frames), and c_(ijk) arethe set of coefficients describing the manner in which the illuminationthat would normally reach each pixel is blocked and/or attenuated at agiven point in time. The noise term, while important to the theory andapplication of compressive sensing, has well understood implications andneed not concern us for purposes of the present discussion. In someimplementations the spatial masking pattern is binary, such that eachc_(ij) value is either 0 or 1, but this is not a necessary constraint.In our example, k ranges from 1 to 100, and i and j each range from 1 to1024. The objective of the mathematical reconstruction, then, is toproduce an estimate of V_(ijk) when M_(ij) and c_(ijk) are known, usingfor example sparsity in some particular mathematical representation toconstrain the underdetermined system of linear equations. Methods fordetermining such mathematical representations and algorithms forperforming the reconstruction are well covered by the (extensive)compressive sensing literature (see for example, Duarte et al. (2008)“Single-Pixel Imaging via Compressive Sampling”, IEEE Signal ProcessingMagazine, March 2008, pages 83-91; Stevens et al., (2015), “ApplyingCompressive Sensing to TEM Video: a Substantial Frame Rate Increase onany Camera”, Adv. Structural and Chemical Imaging 1:10). The process isrepeated for each image M_(ij) returned by the camera, with one M_(ij)recorded per block of time. The reconstruction algorithm can operate ona single M_(ij) at a time, or can operate on multiple M_(ij)simultaneously in order to take advantage of continuity from one set of100 reconstructed frames to the next. Note that, throughout thisdiscussion, the actual physical interpretation of indices i and j willdepend upon the measurement system and its operating mode. In general,they represent the rows and columns of a camera, regardless of how thatcamera is being used. In some cases the camera will be a linear arrayand not a two-dimensional array, and in all such cases the pair ij ofindices should be considered to be replaced by a single index i. In thecase of real-space imaging, the i and j indices will be linearly relatedto the Cartesian coordinates in the plane of the sample or scene understudy. In the case of diffraction patterns, the i and j indices willtypically represent, to a linear approximation, the two-dimensionalscattering angle induced in the probe particles by the sample understudy. In the case of spectroscopy, one of these two indices willrepresent a spectral coordinate (such as energy loss, wavelength shift,or x-ray photon energy) and the other index, if it exists, may or maynot have a simple physical interpretation depending on the physicaloperation principles of the spectroscopy system. For example, inelectron energy-loss spectroscopy this other index typically representsone of the spatial coordinates in the sample plane, one of thecomponents of scattering angle, or a linear combination of these.

The approach to time domain-encoded temporal video compressive sensingdisclosed herein (which can be applicable to more than just videocompressive sensing as it may be applied to other types of data, forexample, spectroscopic results that vary rapidly as a function of time)is mathematically distinct from the spatially-encoded method describedabove. Rather than capture a single image with differentspatially-dependent coefficients that vary in time for each image pixel(or spectroscopy channel, for spectroscopic information), we propose tocapture multiple full resolution images (or, more generally, data sets)per block of acquisition time, each of which is a distinct linearcombination of images from different time slices. Mathematically, thisis represented as:M _(ijl)=Σ_(k) c _(lk) V _(ijk)+noise,  (3)where we have added an additional index 1 to distinguish differentimages (or measurement data sets) acquired during the same dataacquisition period (i.e., the same block of time). Note that thecoefficients c_(lk) are now independent of spatial pixel (i,j). This setc_(lk) of coefficients plays the role of the measurement matrix or maskmatrix Φ, as illustrated for example in FIG. 2.

In one implementation, equation (3) can be interpreted as asserting thatwe have multiple cameras (each with 1024×1024 pixels, for example) and asystem for projecting a different linear combination of time sliceimages onto each such camera, such that it effectively multiplies thecamera speed. The system should be fast enough to switch states manytimes per reconstructed time slice, so that different linearcombinations of each time slice can be sent to each camera. These neednot be physically distinct cameras. They could, for example, be 16distinct regions on a 4096×4096 pixel camera with, for example, afast-switching mirror array (for optical systems) or a high-speeddeflector system (for electron-optical systems) acting as the switchingsystem. If the switching system is extremely fast, then the transients(e.g., blur during the settling time of an electrostatic deflector) maybe negligible on the timescale relevant to the operator. In other cases,it would be advantageous to couple the system with a second high-speedswitching system (e.g., a beam blanker in an electron microscope) thatprevents signal from reaching the detector during this transient time.The switching could also be done with an array of variablebeam-splitting systems that can each send some fraction of signal toeach of two different paths, using for example electro-opticalmodulators. In another implementation, the multiple “cameras” could bemultiple sets of local capacitive bins for storage of intensityinformation in a large and complex complementary metal oxidesemiconductor (CMOS) detector array, with a high-speed clock/multiplexersystem for deciding which set of bins is to be filled at any given pointin time. In all of these cases “fast” and “high-speed” are relative tothe duration of a time slice, such that the system must be able toswitch states multiple times per time slice. Or, if the sequence ofevents represented by the video V_(ijk) is precisely reproducible, eachindex could represent a separate run of this sequence of events with adifferent temporal masking pattern c_(lk) for each, for example byrapidly modulating the electron beam current as a function of time in anelectron microscope during each acquisition. All of these potentialphysical embodiments represent different implementations of the samemathematical model represented in equation (3). Note that in manyembodiments of the disclosed temporal compressive sensing method, thetemporal switching may be accomplished either through the design of theillumination system (to enable rapidly varying illumination intensities)or through the design of the detection system (using multiple sensors ora rapid switching system as described above) while still realizing thesame concept described by the mathematical model.

As used throughout this disclosure, the terms “rapid”, “rapidly”, “fast”and “high-speed” are used to characterize the timescale on whichspecified process steps occur relative to the duration of a dataacquisition period (e.g. the exposure time for an image sensor). Forexample, a “rapid” switching process may be one in which the system iscapable of switching at least 2 times, at least 4 times, at least 6times, at least 8 times, at least 10 times, at least 25 times, at least50 times, at least 75 times, at least 100 times, or more, betweendifferent system states (e.g. states corresponding to differentillumination intensities) during the course of a single data acquisitionperiod (e.g., the exposure interval or data acquisition period used tocapture an image with an image sensor).

In many embodiments, the number of time slices is not dictated by thephysical measurement system itself and can be adjusted after the factduring the computational analysis of the data to allow the effectiveframe rate to be adapted to the data. The compressibility andsignal-to-noise ratio of the data stream may not be known in advance,and may indeed vary with time for a single series of acquisitions. Thecomputer software that performs the reconstruction will know exactlywhich detector(s) or detector region(s) were receiving signal at everysingle point in time during each acquisition and, therefore, thecomputer may calculate a range of measurement matrices, each with adifferent number of time slices. In a non-adaptive system, thesecalculations could be performed before any measurements are acquired,thus saving computation time during the acquisition. Based on any of anumber of readily available mathematical metrics (e.g. the calculatedreconstruction uncertainty in a Bayesian model), the software couldchoose the number of time slices for each acquisition in such a way asto produce a specified level of reconstruction fidelity while stillproviding the highest effective time resolution possible. In the limitof extremely low compressibility of the data stream, such a system mayat times use a number of time slices equal to the number of detectors(or detector regions). This will always be possible provided one definesthe acquisition sequence so that the square measurement matrix producedin this case is sufficiently well-conditioned, allowing numericallystable calculation of the no-longer-underdetermined set of linearequations. Such adaptive reconstruction techniques are not necessarilypossible, or not necessarily as effective or practical or easy tocalculate, in the case of compressive sensing based on spatialmodulation, which requires significant computation to be performedbefore the reconstruction produces even a recognizable image, and in thecase of extremely poor signal-to-noise ratio and excessive compression,may never produce a recognizable image at all.

Computer simulations (e.g., see Example 1) demonstrate that a timedomain-encoded temporal compressive sensing system based on equation (3)can provide reconstruction of video data with the number of time slicessignificantly exceeding the number of measurements (i.e., the number ofdistinct values of the index l), using algorithms similar to thosedescribed in the technical literature (e.g., l₁-norm regularization,total-variation (TV) regularization, and dictionary learning (Bayesianor otherwise)). These results establish the mathematical validity of theconcept, and place it in a position to take advantage of continuedadvances in compressive sensing algorithms.

Temporally multiplexed compressive sensing: A more general model, whichwe will call temporally multiplexed compressive sensing (TMCS), can beconstructed that includes equations (2) and (3) as special cases:M _(ijl)=Σ_(k) c _(ijkl) V _(ijk)+noise,  (4)which can be interpreted in two different ways. We can describe this asmultiple simultaneous (or effectively simultaneous, if we have aswitching system that can change states many times within a single timeslice) measurements of the type described by equation (2) or as ameasurement of the type described by equation (3) but with theadditional flexibility afforded by allowing the c_(ijkl) coefficients tovary as a function of position as well as time. Implementing this in themultiple-capacitive-bin CMOS concept, or in a system based on the use ofa micromirror array, may be quite feasible. The concept described inUnited States Patent Application 2015/0153227A1 implements equation (3)in the limited case of only two distinct values of the index l, as itdescribes two coded-aperture video systems operating in parallel, thuspotentially overcoming some of the mathematical difficulties of videoreconstruction when the measured data are limited to a single codedaperture. This is entirely distinct from the concepts of the presentdisclosure. The concept in U.S. 2015/0153227 A1 still achieves videocompression using the essential modality of other coded-aperture videosystems, and it only uses the redundant measurement to improve themathematical properties of the reconstruction. US20150153227 A1 does notrecognize that, when the number of simultaneously acquired data sets(e.g., full-resolution images) exceeds 2, an entirely different modalityof temporal compression becomes available, as described in the presentdisclosure. The methods and systems of the present disclosure canoperate in the mode described by equation (3), but in many embodimentsthey are not necessarily limited to this mode, for example they canoperate in a mode described by the more general equation (4). Themethods and systems of U.S. 2015/0153227 A1 cannot effectively operatein the mode described by equation (3), for they would be limited to avery small number M=2 of measurements, and sparsity-based reconstructionmethods perform poorly, if at all, for such a small number ofmeasurements. Further, the compressive sensing scheme disclosed in U.S.2015/0153227 A1, like all coded-aperture video compression schemes,requires significant computational resources to produce a reconstructedvideo of acceptable quality. This is because the compression schemeemployed depends on a complicated scheme of spatiotemporal modulation,and coded-aperture schemes only directly capture one (in most cases) ortwo (in the case of U.S. 2015/0153227 A1) actual real-space imagesduring a single block of time. The scheme of the present invention, incontrast, captures multiple full-resolution data sets (e.g., images) ineach block of time, and even an elementary pseudoinverse calculation(which requires a negligible fraction of one second) suffices to providea first-approximation reconstruction that clearly resembles the finalresult well enough for a human user to evaluate the quality of theacquisition in real time. Finally, in many embodiments, the presentlydisclosed methods and systems can direct virtually all of the photons(in an optical system) or electrons (in an electron microscope) to thevarious detectors or detector regions, without significant waste. Bycontrast, coded-aperture schemes by their very nature block substantialfractions of the signal (typically ˜50%).

This concept can be generalized yet further into a model:M _(ijl)=Σ_(i′,j′,k) c _(i,ji′j′kl) V _(i′j′k)+noise,  (5)where the intent is that the index i has the same range as the index i′and the index j has the same range as the index j′. This equationindicates that the measurement M_(ijl) consists of multiple measurementsof images of the same size and shape as the images to be reconstructed,but that the coefficients can now mix information from different partsof the image, for example in order to implement such things asconvolution filters (so that the compressive sensing reconstructionprocess also performs a de-blurring enhancement or an edge-enhancementor some other feature enhancement, for example based on learned oroptimized dictionaries) or complex coding schemes that take advantage ofthe typical patterns of spatiotemporal correlation in a video tominimize redundancy in the extraction of information from the systembeing measured.

Finally, we can remove the constraints on the indices i and j inequation (5) and produce a model in which the measurement is just ageneral linear operator acting on the video (or sensor) data, plus anoise term. If we further eliminate the concept of “blocks of time” sothat (for example) the system operates in a rolling-acquisition modewithout well-defined non-overlapping blocks of time slices, and if weallow the time slices themselves to vary in duration and even topartially overlap, then the model becomes quite general indeed.

With each generalization of the fundamental model, there is thepotential for improving the performance of the compressive sensingreconstruction system, including adding new capabilities such asde-blurring. Generalizing the model certainly cannot make theperformance worse, since by the very nature of generalization eachspecific model is a strict subset of the more general one. Thisgeneralization comes at the cost of complexity (both in the physicalacquisition system required and in the reconstruction algorithm used)and, potentially, the computational resources required for thereconstruction. The real-world value and practicality of implementingthe generalized conceptual models described by equations (4) and (5) canbe assessed through numerical simulations. It is already known fromnumerical simulation that equations (2) and (3) can each form the basisof an effective time domain-encoded compressive sensing system that canbe used to reconstruct significantly more frames of video data (or, moregenerally, time-dependent data sets) than are directly measured. Thereis published work on video compressive sensing usingspatial-multiplexing cameras (SMCs) based, for example, on asingle-pixel camera (see, for example, Duarte et al., “Single-PixelImaging Via Compressive Sampling”, IEEE Signal Processing Magazine,March, 2008, page 83-92), but this is approach is mathematicallydistinct from the TMCS approach disclosed herein which directly capturesmultiple images per block of time with no need for complex encoding orreconstruction of the spatial information.

The compressive sensing system concept disclosed herein is that of asystem that acquires not just one but multiple images (or data sets)from a single block of data acquisition time, with each image or dataset representing a different linear combination of time slices withinthat block of time. These multiple images or data sets compriseintensity data acquired using a system that either simultaneously sendssignal to multiple detectors (e.g., an optical beam splitter array withrapid switching achieved using an electro-optical modulator), or thatselects which detector is to receive the signal at any given instant intime using a switching system (e.g., a set of deflector plates for anelectron microscope) that can switch multiple times per time slice.

Advantages of temporally multiplexed compressive sensing: In addition toovercoming the disadvantages of coded-aperture video compressive sensingthat are specific to electron microscope applications, as discussedabove, TMCS may overcome blurring artifacts associated with opticalcoded-aperture compressive sensing. Coded-aperture compressive sensingcan produce noticeable blurring artifacts aligned with the direction ofmotion of the aperture. While these artifacts are sometimes negligible,there are cases (e.g., in videos of complex scenes in which there aremany objects in motion at speeds comparable to one pixel per time slice,or greater) in which the artifacts are quite obvious. Because the TMCSapproach does not inherently involve any “scrambling” of the spatialinformation or any preferred direction in image space, this particularsource of reconstruction distortion does not exist in TMCS.

In addition, TMCS produces directly interpretable images even before anyreconstruction is applied. Further, unlike a coded-aperture system, aTMCS system can be operated in a mode that produces high-time-resolutionvideos directly, by operating in a direct acquisition mode rather than acompressive-sensing mode. For example, if we have a system that captures16 images per block of time, with an arbitrary (up to the physicallimits of the switching system) coefficient matrix c_(lk) (as describedin Equation (3)), we can, if we wish, specify that some or all of the 16images do not mix information from widely separated points in time, butrather collect data from a contiguous small number (perhaps only one) ofthe time slices. In this case the exposure time for each of the 16images can be extremely short, limited by the speed of the switchingsystem, provided the available illumination intensity is high enough toproduce an image of adequate signal-to-noise ratio in such a short time.Thus the TMCS system could be operated so that some, or even all, of themeasured images represent snapshots with extremely low exposure times,even much shorter than the time slices used in a typical compressivesensing mode. The price of this operation mode, if taken to its limit,is that the duty cycle of the exposure may be extremely low, so thatlittle or no information is available from some, perhaps most, of thetime slices. For some applications (e.g., experiments in which asequence of events is triggered and thus will come at some preciselyknown span of time), this may provide extremely high time resolutionthat is difficult to obtain through other approaches, thus entering anapplication space overlapping with that of Movie Mode DynamicTransmission Electron Microscopy (previously described in U.S. Pat. No.9,165,743).

The simpler mathematical form of the governing equation for TMCS(equation (3)) as opposed to coded-aperture video compressive sensing(equation (2)) can have advantages in terms of the computationalresources required for reconstruction. Because the spatial informationis represented directly in TMCS, a rough-draft reconstruction can beproduced extremely quickly by any of a number of simple algorithms(e.g., placing each acquired image into the span of time slices in whichits coefficients are greater than the coefficients of any other acquiredimage), and iterative algorithms can incrementally improve that estimateboth online (i.e., during the ongoing acquisition) and offline (i.e.,later on, possibly with a much larger computer). Many other compressivesensing systems provide a compressed data stream that cannot be directlyinterpreted, and must go through significant processing beforerecognizable results appear, and this can be a significant problem forpractical implementation, since the user sometimes cannot see whetherthe data is useable until long after the experiment is over.

Provided the switching-time overhead is small (i.e., only a smallfraction of the time is spent switching from one set of output channelsto another), the effective duty cycle of TMCS (i.e., the fraction oftotal available signal that the system acquires) can be very close to 1.Typically, coded-aperture compressive sensing has a duty cycle ofapproximately one half, since roughly half of the pixels are blocked atany given instant in time. This means TMCS can potentially make betteruse of available signal, by nearly a factor of 2.

Applications: The time domain-encoded temporal compressive sensingmethods disclosed herein may be adopted in a variety of imaging andspectroscopy applications including, but not limited to, optical videoimaging, time-resolved optical spectroscopy, and transmission electronmicroscopy (e.g., for capture of image data, diffraction pattern data,electron energy-loss spectra, energy-dispersive X-ray spectra, etc.).Furthermore, the temporal compressive sensing methods disclosed hereinmay be used to capture signals (i.e., images, spectra, diffractionpatterns, etc.) arising through the interaction of radiation with asample or scene such that the radiation is transmitted, reflected,elastically scattered, or in-elastically scattered by the sample orscene, thereby forming patterns of transmitted, reflected, elasticallyscattered, or inelastically scattered radiation which are detected usingone- or two-dimensional sensor arrays. Depending on the application, theradiation may be electro-magnetic radiation, particle radiation, or anycombination thereof. Suitable radiation sources include, but are notlimited to, electromagnetic radiation sources, electron guns, ionsources, particle accelerators, and the like, or any combinationthereof.

The time domain-encoded temporal compressive sensing methods disclosedherein may be directly applied to the study of the evolution of eventsand physical processes in time. However, its range of application goeswell beyond this, because there are numerous applications in whichanother coordinate of interest may, in effect, be mapped to the timeaxis by the manner in which the system works. One such example istomography, in which a sample under study is rotated and a series ofmeasurements is acquired over a range of rotation angles. Rotating thesample implies a varying sample orientation as a function of time, i.e.,a mapping (not necessarily one-to-one) between orientation and time. Incases such that the ability to capture tomographic data is limited bythe measurement rate of a camera, temporal compression couldsubstantially accelerate data acquisition upon increasing the rate ofsample rotation to take advantage of the increased effective frame rateof the camera. Similarly, scanning transmission electron microscopy(STEM) operates by scanning a focused electron beam across a region of asample (i.e., in which the electron beam diameter is narrow relative tothe cross-sectional area of the sample to be analyzed or imaged) andcapturing a data set (be it a high-angle annular dark field (HAADF)signal, an electron energy-loss spectrum (EELS), an energy-dispersivex-ray spectrum (EDX), a bright-field signal, a diffraction pattern, or acombination of these) at every scan position. The act of scanningcreates a mathematical map between position and time and, just as in thetomography example, if the system limitation is in the camera speed (asit very often is, for example, in STEM-diffraction), then temporalcompression has the potential to greatly improve data throughput. Thisembodiment would have similar capabilities as the methods and systemsdisclosed in U.S. Pat. No. 9,165,743, but it operates on a completelydifferent principle. Specifically, the presently disclosed methodsachieve compressive sensing primarily through temporal modulation (inthis case, by varying the position on the sample of the focused electronprobe as a function of time) and, while they may take advantage ofspatial modulation, they are not necessarily dependent on spatialmodulation. All previous applications of compressive sensing in electronmicroscopy, both proposed and actually implemented, necessarily rely oneither spatial modulation or simple under-sampling and in-painting toachieve compression, and fail to describe the mechanism of temporalcompression described in the present disclosure. As illustrated in thetomography and scanning transmission electron microscopy (STEM) examplesdiscussed above, in some embodiments of the disclosed temporalcompressive sensing methods and systems, distinct linear combinations ofpatterns of the radiation transmitted, reflected, elastically scattered,or in-elastically scattered by a sample (or a scene) for a series oftime slices may be generated by modulating an experimental parameterother than the radiation intensity itself in a temporal fashion. Forexample, in some embodiments, the experimental parameter to betemporally modulated may be selected from the group consisting ofrotational orientation of the sample, linear translation and/or tilt ofthe electron probe in one dimension, linear translation and/or tilt ofthe electron probe in two dimensions, linear translation of the samplein one dimension, linear translation of the sample in two dimensions,and linear translation of the sample in three dimensions, or anycombination thereof. In some embodiments, the radiation incident on thesample (or scene) is focused to a narrow beam (i.e., having a beamdiameter that is small relative to the cross-sectional area of thesample or scene to be imaged or analyzed) and the experimental parameterto be temporally modulated is the position of the beam relative to thesample (or vice versa).

Optical imaging & spectroscopy systems: Optical imaging and spectroscopysystems based on the disclosed time domain-encoded temporal compressivesensing may be developed for a variety of applications using a varietyof commercially-available optical system components, e.g., lightsources, optical modulators, and sensors, as well as other active orpassive components such as lenses, mirrors, prisms, beam-splitters,optical amplifiers, optical fibers, optical filters, monochromators,etc. Examples of optical imaging applications include, but are notlimited to, video imaging, visible light imaging, infrared imaging,ultraviolet imaging, fluorescence imaging, Raman imaging, and the like.Example of spectroscopy applications include, but are not limited to,absorbance measurements, transmittance measurements, reflectancemeasurements, fluorescence measurements, Raman scattering measurements,and the like.

Light sources for use in temporal compressive sensing systems of thepresent disclosure may include, but are not limited to, incandescentlights, tungsten-halogen lights, light-emitting diodes (LEDs), arclamps, diode lasers, and lasers, or any other source of electromagneticradiation, including ultraviolet (UV), visible, and infrared (IR)radiation. In some applications, natural light arising from solarradiation (i.e., produced by the sun), may serve to illuminate a sampleor scene for which temporally compressed data is acquired.

High speed switching of optical signals may be achieved through any of avariety of approaches including, but not limited to, the use of opticalmodulators, e.g., electro-optic modulators or acousto-optic modulators,or digital micro-mirror array devices. In some embodiments of thedisclosed compressive sensing methods and systems, the switching timesachieved may range from less than 1 nanosecond to about 10 milliseconds.In some embodiments, the switching times may be at least or at leastabout 1 nanosecond, at least or at least about 10 nanoseconds, at leastor at least about 100 nanoseconds, at least or at least about 1microsecond, at least or at least about 10 microseconds, at least or atleast about 100 microseconds, at least or at least about 1 millisecond,or at least or at least about 10 milliseconds. In some embodiments, theswitching times achieved may be at most or at most about 10milliseconds, at most or at most about 1 millisecond, at most or at mostabout 100 microseconds, at most or at most about 10 microseconds, atmost or at most about 1 microsecond, at most or at most about 100nanoseconds, at most or at most about 10 nanoseconds, or at most or atmost about 1 nanosecond. Those of skill in the art will recognize thatthe switching times that are achievable may have any value within thisrange, e.g. about 500 nanoseconds.

Examples of suitable sensors, sensor arrays, or detectors for use in thetemporal compressive sensing methods of the present disclosure include,but are not limited to, photodiodes, avalanche photodiodes, photodiodearrays, photomultipliers, photomultiplier arrays, charge coupled devices(CCDs), image intensified CCDs, and complementary metal oxidesemiconductor (CMOS) sensors, CMOS framing cameras (e.g., CMOS camerasthat can store multiple images or datasets on-chip through the use ofmultiple capacitive bins at each pixel and an electronic switchingsystem that determines which set of bins is accumulating signal at anygiven time), or any combination thereof. In some embodiments, thesensors, sensor arrays, or detectors for use in the temporal compressivesensing methods of the present disclosure may further comprise anonlinear optical material, a fluorescent material, a phosphorescentmaterial, or a micro-channel plate, that converts or amplifies theradiation provided by the radiation source into a form of radiation thatis directly detectable by the sensor, sensor array, or detector. Forpurposes of the present disclosure, the term “sensor array” and itsgrammatical equivalents is meant to include “point” arrays (e.g., singlepixel sensors) as well as one-dimensional (linear) arrays,two-dimensional arrays, and so forth. Furthermore, the term “detector”and its grammatical equivalents is meant to include the individualsensors and sensor arrays, as described above, as well as combinationsof optical components and sensors, for example, spectrometers comprisinga monochromator optically coupled with a photodiode array or CCD camera.Suitable linear or two-dimensional sensor arrays may comprise a widevariety of individual pixels.

Sensor arrays suitable for use in the disclosed temporal compressivesensing systems may comprise from or from about 2 to 100×10⁶ pixels, ormore. In some embodiments, sensor arrays for use in the disclosedtemporal compressive sensing systems may comprise at least or at leastabout 2 pixels, at least or at least about 10 pixels, at least or atleast about 100 pixels, at least or at least about 1,000 pixels, atleast or at least about 10,000 pixels, at least or at least about100,000 pixels, at least or at least about 1,000,000 pixels, at least orat least about 10×10⁶ pixels, at least or at least about 100×10⁶ pixels,or more. In some embodiments, sensor arrays for use in the disclosedtemporal compressive sensing systems may comprise at most or at mostabout 100×10⁶ pixels, at most or at most about 10×10⁶ pixels, at most orat most about 1,000,000 pixels, at most or at most about 100,000 pixels,at most or at most about 10,000 pixels, at most or at most about 1,000pixels, at most or at most about 100 pixels, at most or at most about 10pixels, or at most or at most about 2 pixels. One of skill in the artwill recognize that the total number of pixels in the sensor array mayinclude any value within this range, for example, about 12×10⁶ pixels.

The term “about” and its grammatical equivalents, in relation to areference numerical value can include a range of values plus or minus10% from that value. For example the amount “about 10” can includeamounts from 9 to 11. The term “about” in relation to a referencenumerical value can also include a range of values plus or minus 10%,9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, or 1% from that value.

Sensor arrays suitable for use in the disclosed temporal compressivesensing systems may comprise pixels of size ranging from or from about0.1 μm to or to about 20 μm on a side. In some embodiments, sensorarrays for use in the disclosed temporal compressive sensing systems maycomprise pixels of at least or at least about 0.1 μm, at least or atleast about 0.25 μm, at least or at least about 0.5 μm, at least or atleast about 0.75 μm, at least or at least about 1 μm, at least or atleast about 2.5 μm, at least or at least about 5 μm, at least or atleast about 7.5 μm, at least or at least about 10 μm, at least or atleast about 15 μm, or at least or at least about 20 μm, or larger. Insome embodiments, sensor arrays for use in the disclosed systems maycomprise pixels of at most or at most about 20 μm, at most or at mostabout 15 μm, at most or at most about 10 μm, at most or at most about7.5 μm, at most or at most about 5 μm, at most or at most about 2.5 μm,at most or at most about 1 μm, at most or at most about 0.75 μm, at mostor at most about 0.5 μm, at most or at most about 0.25 μm, or at most orat most about 0.1 μm on a side, or smaller. One of skill in the art willrecognize that the pixels in the sensor array may have any value withinthis range, for example, about or at most about 0.8 μm on side.

Sensor arrays suitable for use in the temporal compressive sensingsystems of the present disclosure may operate at data acquisition andread-out rates ranging from or from about 0.001 frames/sec (or lower) toor to about 100,000 frames/sec (or higher). In some embodiments, sensorarrays suitable for use in the disclosed temporal compressive sensingsystems may operate at data acquisition and read-out rates of at leastor at least about 0.001 frames/sec, at least or at least about 0.01frames/sec, at least or at least about 0.1 frames/sec, at least or atleast about 1 frame/sec, at least or at least about 10 frames/sec, atleast or at least about 100 frames/sec, at least or at least about 1,000frames/sec, at least or at least about 10,000 frames/sec, at least or atleast about 100,000 frames/sec, or higher. In some embodiments, sensorarrays suitable for use in the disclosed temporal compressive sensingsystems may operate at data acquisition and read-out rates of at most orat most about 100,000 frames/sec, at most or at most about 10,000frames/sec, at most or at most about 1,000 frames/sec, at most or atmost about 100 frames/sec, at most or at most about 10 frames/sec, atmost or at most about 1 frame/sec, at most or at most about 0.1frames/sec, at most or at most about 0.01 frames/sec, or at most or atmost about 0.001 frames/sec, or lower. One of skill in the art willrecognize that the sensor array may operate at a data acquisition andread-out rate having any value within this range, for example, about 60frames/sec.

For temporal compressive sensing systems in which high speed switchingcomponents are used to deflect images or other datasets to one ofseveral different regions (or “sub-regions”, “sub-units”, etc.) of atwo-dimensional sensor array, the total number of available regions maycomprise either a linear array or a two dimensional array comprisinganywhere from 2 to 400 or more individual regions. For two dimensionalsensor arrays in which the pattern of regions is organized as a squareN×N array, the array of regions may comprise a 2×2 array, a 3×3 array, a4×4 array, a 5×5 array, a 6×6 array, a 7×7 array, an 8×8 array, a 9×9array, a 10×10 array, an 11×11 array, a 12×12 array, a 13×13 array, a14×14 array, a 15×15 array, a 16×16 array, a 17×17 array, an 18 x 18array, a 19×19 array, or a 20×20 array, or a higher dimension N×N array.In some embodiments, the pattern of regions may be organized as arectangular array (e.g., an M×N array) comprising a 2×3, array, a 2×4array, a 2×5 array, a 2×6 array, a 3×2 array, a 3×4 array, a 3×5 array,a 3×6 array, a 4×2 array, a 4×3 array, a 4×5 array, a 4×6 array, a 5×2array, a 5×3 array, a 5×4 array, a 5×6 array, a 6×2 array, a 6×3 array,a 6×4 array, a 6×5 array, or a higher order M×N array. In someembodiments, the pattern of regions may comprise a hexagonal array, aparallelogram array, an irregular array, a randomly distributed array,or any combination thereof, with or without missing elements. Eachregion may have no overlap with other regions, or some regions may havepartial or full overlap with some regions, or some regions may besubsets of other regions. Each region may be a circular region, anelliptical region, a square region, a rectangular region, a hexagonalregion, a regular polygonal region, an irregular polygonal region, aregion of any shape comprising a simply-connected subset of pixels, or aregion of any shape comprising a non-simply-connected subset of pixels.Each region may be identical in size and shape to all other regions, orsome regions may differ in size from other regions, or some regions maydiffer in shape from other regions, or some regions may differ in bothsize and shape from other regions. Each region may be identical inorientation to other regions, or some regions may have orientationsrotated with respect to the orientations of other regions, or someregions may have orientations reflected with respect to the orientationsof other regions, or some regions may have orientations that are bothrotated and reflected with respect to the orientations of other regions.Each region may be identical in scale or magnification to other regions,or some regions may differ in scale or magnification in one coordinateaxis with respect to other regions, or some regions may differ in scaleor magnification in two coordinate axes with respect to other regions.In the case in which one or both coordinates in the camera plane may beidentified with real-space coordinates in a sample plane or a scene,each region may record the same region of such real-space coordinates asother regions, or the regions it records may partially overlap with, orbe a strict subset of, or be a strict superset of, one or more othersuch regions. In the case in which one or both coordinates in the cameraplane may be identified with a linear approximation of scatteringangles, each region may record the same set of scattering angles asother regions, or the regions it records may partially overlap with, orbe a strict subset of, or be a strict superset of, one or more othersuch sets of scattering angles. In the case in which one or bothcoordinates in the camera plane may be identified with a spectralcoordinate, including but not limited to energy loss, wavelength shift,or photon energy, each region may record the same region of suchspectral coordinates as other regions, or the regions it records maypartially overlap with, or be a strict subset of, or be a strictsuperset of, one or more other such regions of spectral coordinates.

Electron microscopy & spectroscopy systems: Electron microscopy systemsfor implementing the temporal compressive sensing methods disclosedherein may comprise a variety of system components including, but notlimited to, electron beam sources, electron beam shutters (“beamblankers” or “beam blanking systems”), electron focusing optics, sampleholders that incorporate various sample stimulus devices, electrondeflector systems, and image sensors or other data capture devices.

Suitable electron beam sources may include, but are not limited to,electron guns (electron emitters) based on thermionic, photocathode,laser-driven photocathode, cold emission, or plasma source emissionmechanisms that emit either continuous or pulsed streams of electrons.An exemplary system for generating precisely-controlled series ofelectron pulses is based on the use of an arbitrary waveform generator(AWG) laser system and photocathode, as described in U.S. Pat. No.9,165,743. Electron beam focusing may be achieved in these systemsthrough purely electrostatic approaches and/or may utilize magneticfields.

In some embodiments, the electron microscope system may incorporate asample holder and a sample stimulus mechanism, e.g., a pulsed sampledrive laser that provides highly precise, adjustable, and intense heatfor initiating dynamic processes in the sample under study. Othermethods of initiating processes in the sample may also be employed,e.g., through electrically triggered sample holders, or externalelectronics connected to sample holders that may deliver a voltagepulse, a current pulse, an electrically-driven heat pulse, or an impulsedelivered to the sample with the aid of a nano-indentation device ormicro- or nano-electromechanical system.

In some embodiments, the electron microscope system may incorporateaccurately-timed, high-speed electron deflector systems, includingelectrostatic deflector systems and/or magnetic deflector systems. Anexemplary electrostatic deflector system is described in U.S. Pat. No.9,165,743. One embodiment of an electrostatic deflector system disclosedtherein includes four high voltage switches connected to customizeddeflector plates which are inserted into the lower part of the projectorlens (e.g., the last electromagnetic lens in a standard TEM) below thesample. The two pairs of orthogonally positioned deflector platesdeflect each image (or diffraction pattern, etc.) arising throughinteraction of electrons with the sample to a different part of thecamera, thereby overcoming a typical camera's multisecond refresh rate.Each of the four plates may independently carry a voltage ranging fromor from about, for example, −800V to +800V, thereby allowing completeflexibility over the electron deflection in two dimensions. The cameraitself is typically positioned at or at about 50 cm below this set ofdeflectors, so that the electron beam can be directed to any part of thecamera (e.g., a CCD camera). The space between the deflector plates andthe projector lens pole piece is partially filled with a ceramicmounting, alignment, and electrical connection system integrated withthe deflector plates. Other positions for the deflector system, forexample within an intermediate lens system or inserted through a port inthe TEM's camera chamber, are also possible. The deflector system candirect each of the images (or diffraction patterns, etc.) arising frominteraction with the electron beam to a different region on a largecamera (e.g., a CCD camera), thereby spatially separating the variousimages (or diffraction patterns, spectra, etc.) captured. The imageproduced by the camera then consists of an array (typically 2×2, 3×3,4×4, 5×5, or higher dimensional or non-square array as described above)of images (or diffraction patterns, spectra, etc.) captured fromdifferent points in time.

Examples of suitable sensors, sensor arrays, detectors, or other datacapture devices for electron microscope systems of the presentdisclosure include, but are not limited to, CCD cameras, intensified CCDcameras, CMOS image sensors, direct detection cameras (e.g., CMOSframing cameras that incorporate multiple capacitive bins for each pixeland electronic switching systems that determine which set of bins isaccumulating signal at any given time), electron energy lossspectrometers (e.g. a post-column imaging filter with a CCD camera),energy-dispersive x-ray spectrometers (e.g., a silicon drift detectorplaced near the sample), and the like. In some embodiments, the sensors,sensor arrays, or detectors for use in the temporal compressive sensingmethods of the present disclosure may further comprise a nonlinearoptical material, a fluorescent material, a phosphorescent material, ora micro-channel plate, that converts or amplifies the primary radiationprovided by the radiation source (e.g., electrons) into a form ofradiation that is directly detectable by the sensor, sensor array, ordetector.

Mathematical algorithms for sampling and reconstruction: Mathematicalreconstruction of the “time slice” images or datasets obtained using thedisclosed compressive sensing (sampling) methods may be accomplishedthrough the use of a variety of optimization algorithms designed topenalize non-sparse solutions of an underdetermined system of linearequations via the l_(l) norm, the total number of non-zero coefficients,total variation, or beta process priors; an iterative greedy recoveryalgorithm; a dictionary learning algorithm; a stochastic Bayesianalgorithm; a variational Bayesian algorithm; or any combination thereof.These algorithms vary dramatically in their details and implementations,and undoubtedly new such algorithms shall be introduced frequently inthe literature, but they all fall under the following generaldescription: algorithms for solving, or approximately solving, anunderdetermined system of linear equations through the use of priorknowledge or belief that the solution is sparse or compressible in somemathematical representation, be it a representation that is known apriori, one that is purely learned from the data, or a combination ofthe two.

“l₁-optimization” refers to finding the minimum l₁-norm solution to anunderdetermined linear system of equations, where the l₁-norm is the“size” of the solution vector of the linear system (i.e., the sum of theabsolute values of the solution vector components) in a particularbasis, for example a discrete cosine transform basis, a wavelet basis, acurvelet basis, a noiselet basis, a learned-dictionary basis, or anyother basis, overcomplete or otherwise, that has been shown to inducesparse or approximately sparse representations of realistic data. It hasbeen shown in compressive sensing theory that the minimum l₁-normsolution is also the sparsest possible solution under quite generalconditions (Candés, E., & Romberg, J. (2005). “l₁-magic: Recovery ofSparse Signals via Convex Programming”. URL:www.acm.caltech.edu/11magic/downloads/11magic.pdf, 4, 14; D. Donoho(2006), “For Most Large Underdetermined Systems of Linear Equations theMinimal l₁-norm Near Solution Approximates the Sparest Solution”,Communications on Pure and Applied Mathematics 59:907-934). Moregenerally, the l₁-norm in a particular basis can be used as a penalty orregularization term in a scheme that solves the underdetermined linearsystem of equations to within a specified error term. Often anadditional penalty term involving the “total variation” (TV) is used forimage data, in the context of both exact solutions and approximatesolutions of the underdetermined set of linear equations. TV istypically defined as the sum of the magnitudes (typically either thel₁-norm or the l₂-norm; different authors use different definitions) ofthe intensity gradient vectors calculated at each point in the image.While TV is in general not technically an l₁-norm, its mathematicalbehavior is similar to that of the l₁-norm applied to the full set ofintensity gradients, and as such a TV penalty term tends to favor asparse intensity gradient in the solution. In other words it provides analgorithmic way to introduce a prior expectation that the gradient issparse. This has the effect of reducing noise and favoring solutionsthat resemble relatively uniform regions with sharp, clearly-definedboundaries. As one of many possible examples, one may endeavor tominimize the sum of three terms: the l₁-norm in adiscrete-cosine-transform basis, a term proportional to TV, and a termproportional to the l₂-norm of the error associated with the approximatesolution of the underdetermined linear system of equations. It has longbeen known (Candés, E., & Romberg, J. (2005)) that commonly availablecomputer algorithms, for example those associated with linearprogramming, can solve optimization problems of this general typeefficiently.

Greedy algorithms are iterative approaches to solving systems ofequations where a locally-optimal choice of candidate solutions is madeat each step of the iteration based on a predefined selection rule andthe addition of one of a limited set of candidate solutions to thecurrently existing solution. Often, a greedy algorithm will yield alocally-optimal solution that approximates a globally-optimal solutionin a reasonable amount of computation time. See, for example, Cormen etal., “Greedy Algorithms”, Chapter 16 in Introduction to Algorithms,Third Edition, MIT Press, Cambridge, Mass., 2009, for a more detaileddescription.

Dictionary learning approaches entail developing a “trainingdata”-dependent transform (or dictionary) for which the solutioncoefficients are sparse and the basis vectors need not be orthogonal,which then allows one to solve the linear problem for a given test setof measurements. See, for example, Kreutz-Delgado et al. (2003)“Dictionary Learning Algorithms for Sparse Representation”, NeuralComput. 15(2): 349-396, for a more detailed description. Some algorithmsallow the dictionary to be learned directly from the compressivelysensed data, with no explicit training data. Many such algorithms allowthe dictionary to be refined as additional data come in. In many casesthe dictionary is over-complete, i.e. there are more dictionary elementsthan there are dimensions in the vector space which the dictionary ismeant to represent. Sparsity may still be induced reliably in suchover-complete representations, for example through the use of Bayesianalgorithms using beta process priors; see, for example, J. Paisley andL. Carin, “Nonparametric factor analysis with beta process priors,”International Conference on Machine Learning (ICML), Montreal, Canada,2009. We note that terminology varies; in many contexts, by definition,an over-complete dictionary is not technically referred to as a “basis,”but the term “over-complete basis” is relatively common in thecompressive sensing and machine learning literature. Thus for simplicityof communication in the present context we choose to use this term.

More complete descriptions of these and other algorithms forreconstructing images or other data sets from a set of measurementsacquired using compressed sensing are readily available in the technicalliterature, see for example, Duarte et al. (2008) “Single-Pixel Imagingvia Compressive Sampling”, IEEE Signal Processing Magazine, March 2008,pages 83-91; and Stevens et al., (2015), “Applying Compressive Sensingto TEM Video: a Substantial Frame Rate Increase on any Camera”, Adv.Structural and Chemical Imaging 1:10.

Computer Systems

The present disclosure provides computer control systems that areprogrammed to implement methods of the disclosure. FIG. 9 shows acomputer system 901 that includes a central processing unit (CPU, also“processor” and “computer processor” herein) 905, which can be a singlecore or multi core processor, or a plurality of processors for parallelprocessing, and may include one or more graphics processing units (GPU),or GPU-like parallel computing components, or quantum-computingcomponents or optical computing components or electro-optical computingcomponents. The computer system 901 also includes memory or memorylocation 910 (e.g., random-access memory, read-only memory, flashmemory), electronic storage unit 915 (e.g., hard disk), communicationinterface 920 (e.g., network adapter) for communicating with one or moreother systems, and peripheral devices 925, such as cache, other memory,data storage and/or electronic display adapters. The memory 910, storageunit 915, interface 920 and peripheral devices 925 are in communicationwith the CPU 905 through a communication bus (solid lines), such as amotherboard. The storage unit 915 can be a data storage unit (or datarepository) for storing data. The computer system 901 can be operativelycoupled to a computer network (“network”) 930 with the aid of thecommunication interface 920. The network 930 can be the Internet, aninternet and/or extranet, or an intranet and/or extranet that is incommunication with the Internet. The network 930 in some cases is atelecommunication and/or data network. The network 930 can include oneor more computer servers, which can enable distributed computing, suchas cloud computing. The network 930, in some cases with the aid of thecomputer system 901, can implement a peer-to-peer network, which mayenable devices coupled to the computer system 901 to behave as a clientor a server.

The CPU 905 can execute a sequence of machine-readable instructions,which can be embodied in a program or software. The instructions may bestored in a memory location, such as the memory 910. The instructionscan be directed to the CPU 905, which can subsequently program orotherwise configure the CPU 905 to implement methods of the presentdisclosure. Examples of operations performed by the CPU 905 can includefetch, decode, execute, and write back.

The CPU 905 can be part of a circuit, such as an integrated circuit. Oneor more other components of the system 901 can be included in thecircuit. In some cases, the circuit is an application specificintegrated circuit (ASIC).

The storage unit 915 can store files, such as drivers, libraries andsaved programs. The storage unit 915 can store user data, e.g., userpreferences and user programs. The computer system 901 in some cases caninclude one or more additional data storage units that are external tothe computer system 901, such as located on a remote server that is incommunication with the computer system 901 through an intranet or theInternet.

The computer system 901 can communicate with one or more remote computersystems through the network 930. For instance, the computer system 901can communicate with a remote computer system of a user. Examples ofremote computer systems include personal computers (e.g., portable PC),slate or tablet PC's (e.g., Apple® iPad, Samsung® Galaxy Tab),telephones, Smart phones (e.g., Apple® iPhone, Android-enabled device,Blackberry®), or personal digital assistants. The user can access thecomputer system 901 via the network 930.

Methods as described herein can be implemented by way of machine (e.g.,computer processor) executable code stored on an electronic storagelocation of the computer system 901, such as, for example, on the memory910 or electronic storage unit 915. The machine executable or machinereadable code can be provided in the form of software. During use, thecode can be executed by the processor 905. In some cases, the code canbe retrieved from the storage unit 915 and stored on the memory 910 forready access by the processor 905. In some situations, the electronicstorage unit 915 can be precluded, and machine-executable instructionsare stored on memory 910.

The code can be pre-compiled and configured for use with a machinehaving a processer adapted to execute the code, or can be compiledduring runtime, or can be interpreted from source code during runtimewithout an explicit compilation step, or any combination thereof. Thecode can be supplied in a programming language that can be selected toenable the code to execute in a pre-compiled or as-compiled fashion.

Aspects of the systems and methods provided herein, such as the computersystem 901, can be embodied in programming. Various aspects of thetechnology may be thought of as “products” or “articles of manufacture”typically in the form of machine (or processor) executable code and/orassociated data that is carried on or embodied in a type of machinereadable medium. Machine-executable code can be stored on an electronicstorage unit, such as memory (e.g., read-only memory, random-accessmemory, flash memory) or a hard disk. “Storage” type media can includeany or all of the tangible memory of the computers, processors or thelike, or associated modules thereof, such as various semiconductormemories, tape drives, disk drives and the like, which may providenon-transitory storage at any time for the software programming. All orportions of the software may at times be communicated through theInternet or various other telecommunication networks. Suchcommunications, for example, may enable loading of the software from onecomputer or processor into another, for example, from a managementserver or host computer into the computer platform of an applicationserver. Thus, another type of media that may bear the software elementsincludes optical, electrical and electromagnetic waves, such as usedacross physical interfaces between local devices, through wired andoptical landline networks and over various air-links. The physicalelements that carry such waves, such as wired or wireless links, opticallinks or the like, also may be considered as media bearing the software.As used herein, unless restricted to non-transitory, tangible “storage”media, terms such as computer or machine “readable medium” refer to anymedium that participates in providing instructions to a processor forexecution.

Hence, a machine readable medium, such as computer-executable code, maytake many forms, including but not limited to, a tangible storagemedium, a carrier wave medium or physical transmission medium.Non-volatile storage media include, for example, optical or magneticdisks, such as any of the storage devices in any computer(s) or thelike, such as may be used to implement the databases, etc. shown in thedrawings. Volatile storage media include dynamic memory, such as mainmemory of such a computer platform. Tangible transmission media includecoaxial cables; copper wire and fiber optics, including the wires thatcomprise a bus within a computer system. Carrier-wave transmission mediamay take the form of electric or electromagnetic signals, or acoustic orlight waves such as those generated during radio frequency (RF) andinfrared (IR) data communications. Common forms of computer-readablemedia therefore include for example: a floppy disk, a flexible disk,hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD orDVD-ROM, any other optical medium, punch cards paper tape, any otherphysical storage medium with patterns of holes, a RAM, a ROM, a PROM andEPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wavetransporting data or instructions, cables or links transporting such acarrier wave, or any other medium from which a computer may readprogramming code and/or data. Many of these forms of computer readablemedia may be involved in carrying one or more sequences of one or moreinstructions to a processor for execution.

The computer system 901 can include or be in communication with anelectronic display 935 that comprises a user interface (UI) 940.Examples of UI's include, without limitation, a graphical user interface(GUI) and web-based user interface.

Methods and systems of the present disclosure can be implemented by wayof one or more algorithms. An algorithm can be implemented by way ofsoftware upon execution by the central processing unit 905.

EXAMPLES Example 1 Computer Simulations

Computer simulations demonstrate that a time domain-encoded temporalcompressive sensing system based on the model described by equation (3)can provide reconstruction of video data with the number of time sliceimages significantly exceeding the number of measurement images. FIG. 1shows 10 frames of TEM image data from an in situ tensile crackpropagation experiment (courtesy K. Hattar et al., Sandia NationalLaboratory). Different combinations of the ten time slice images(illustrated schematically in FIG. 2) are sent to four different regionson a large area camera, for example, by using a fast beam deflectionsystem installed in the TEM, and digitally segmented into fourmeasurement image frames for analysis (FIG. 3). In this non-limitingexample, the fast beam deflection system provides the ability to acquire4 measurement image frames in one camera data acquisition period (i.e.,during a single exposure). 16-frame fast deflector systems are alreadyavailable, and compression factors much greater than the 10/4=2.5 valueillustrated in this example are expected to be achievable. Applicationof sparse mathematical reconstruction techniques to the four measuredimage frames provides a reliable estimate of all ten time slice frames(FIG. 4). The same algorithm captures subtle details (e.g., changes indiffraction contrast in the stress-concentration region before failure)as well as gross discontinuities (e.g., the sudden change from timeslice 7 to time slice 8). The simulation results demonstrate thereconstruction of 10 frames of video data from a single exposure period.Use of a 16 frame fast deflector system (i.e., one that captures 16segmented image frames per camera data acquisition period) andapproximately 6× compressibility, would provide approximately 100 framesof reconstructed video data per single exposure.

Example 2 TEM-Based Temporal Sensing System Using Post-Sample Deflector

As an illustrative (prophetic) example, consider a TEM with a rapid,post-sample deflector system, a relatively large camera (e.g., a CCDcamera with a scintillator and fiber-optic bundle, as is commonly usedfor TEM data acquisition), and an optional pre-sample beam blankingsystem. FIG. 5 shows a generic, simplified schematic of the basiccomponents and function of a TEM. The electron source produces a beam ofelectrons which are accelerated to kinetic energies of typically ˜80 keVto ˜300 keV per electron for most current instruments. A condenser lenssystem focuses a selected part of the electron beam onto a sample placednear the center of an objective lens. The beam passes through thesample, and the intermediate/projector lens system produces either animage or a diffraction pattern that can be captured by a dataacquisition system. The data acquisition system is typically either acamera or a post-column energy-filter system that itself includes acamera. The energy-filter system adds energy-filtered acquisition andelectron energy-loss spectroscopy (EELS) capabilities to the system.Other systems (e.g., in-column energy filters) exist that producesimilar results. The acquisition system includes a detector, typicallybut not necessarily either a CCD camera with a scintillator or adirect-detection CMOS camera or similar technology. The data acquisitionrate of the system is therefore set by the acquisition and readout timeof the camera (henceforth “data acquisition period” or “camera frametime”).

FIG. 6 illustrates one non-limiting example of a TEM system thatutilizes a high-speed deflector system positioned after the sample(shown here, for example, positioned after the projector lens system)that allows multiple distinct frames to be directed to (preferably, butnot necessarily) non-overlapping regions of a large area camera.Operation is not fundamentally changed for a post-column energy-filteredimaging system. “High speed” in this context means the deflector canswitch states many times (at least about 10 times, although preferablyhundreds or thousands of times) per camera data acquisition period whileintroducing negligible blur. An optional high-speed beam blanker candirect the beam to an aperture while the high-speed deflector state isswitching, for example positioned high in the condenser lens system (asshown), in order to reduce or eliminate blur effects.

We anticipate that a system that can switch states among a 2×2, 3×3, or4×4 (or higher order) array of camera sub-regions with ˜10 ns spentduring each switching operation and ˜100-1000 switching operations percamera frame time. This allows each of 10-100 or more “time slices” percamera frame time to be represented, in part, in multiple sub-regions ofthe camera. Mathematically, this is represented as a measurement matrixthat tracks how each sub-region (or “measured frame”) represents adifferent linear combination of time slices. The mathematical techniquesassociated with compressive sensing can then produce reliable estimatesof all of the individual time slice datasets, with the result that ˜100distinct data frames are captured in a single camera data acquisitionperiod.

In the case of an imaging filter, the “effective camera area” is not tobe interpreted as the literal camera position but rather as an objectplane that is coupled to an image plane at the actual physical cameraposition.

In some embodiments, instead of a rapid deflector, the system mayinclude a solid-state multi-frame detection system, e.g., a CMOS-arrayframing camera having multiple storage bins per detection pixel and theability to arbitrarily (or semi-arbitrarily) control which set ofstorage bins are accumulating signal at any given moment in time.Functionally the result is nearly the same; this embodiment justreplaces the multi-frame switching capability of the electron opticswith the multi-frame switching capability of the detector. Depending onthe design of the sensor chip, such a system could operate on themathematical model of equation (3), equation (4), or equation (5).

The deflector system illustrated in FIG. 6 may be installed after theprojector lens in a TEM, for example using existing camera/detectorports. Using existing ports allows the modification to be quitenon-invasive, comparable to the installation of cameras and otherdetectors, and the resulting system will not interfere with normaloperation since the deflector can be easily retracted. As describedabove, the deflector is designed to laterally deflect the TEM image toany of several sub-regions of the camera's imaging sensor, for example,to any sub-region in a 4×4 array of 16 sub-regions, similar to thedeflector system described for a Movie Mode Dynamic TransmissionElectron Microscope (see U.S. Pat. No. 9,165,743). The deflector ispreferably electrostatic rather than electromagnetic, thereby allowingexisting circuit designs to switch discretely from one sensor arraysub-region to another in roughly 10 nanoseconds. If the system is usedto reconstruct a video with, for example, 10 μs time slices and using˜10 deflections per time slice, the duty cycle for the system is ˜99%,and the blurred images from the remaining 1% of electrons should notsubstantially interfere with the CS reconstruction algorithms. Forshorter time slices, it may be desirable to also insert a high-speedelectrostatic beam blanker before the sample, thus shutting off theelectron beam during the transitions and eliminating this source ofblurring. For typical TEM electron gun and condenser lens systemdesigns, the time resolution of such a system would be determined moreby available beam currents and acceptable signal-to-noise ratios than bythe time resolution of the deflection system itself. Thus the systemwould also benefit from other modifications to increase the beam currentthat can be delivered to the sample. Note that although this discussionhas focused on real-space imaging, all TEM implementations discussedabove and elsewhere in this disclosure can potentially be used fordiffraction or spectroscopy as well.

Example 3 Optical Temporal Sensing System with Multiple Cameras & EOMSwitching

As another illustrative (prophetic) example, consider a set of opticalcameras with an electro-optic modulator-controlled switching network, asillustrated in FIG. 8. Electro-optic modulators (EOMs) and other highspeed modulators (for example acousto-optic modulators (AOMs)) can beused to rapidly switch an optical signal between two different outputpaths. This switching could be implemented in a binary fashion (suchthat the signal goes to only one of the two output paths) or incontinuous fashion (with the ability to control the fraction of signalto be sent to each output path). A network of such switches could leadto an array of detectors, each of which is a full resolution camera (orspectroscopic system) in its own right. While the engineering complexityof designing such a system for real-space imaging may be high,implementation in the field of time-resolved spectroscopy may be easierby taking advantage of well-developed EOM/AOM solutions for fiber-opticsystems. A network of optical fibers and modulators would feed aparallel array of spectrometers (or a single spectrometer with a largetwo-dimensional sensor that can act, in effect, as a parallel array),and an electronic control system would determine what superposition oftime slices is sent to each individual spectrometer. This optical systemcould operate in either a single-shot or a stroboscopic mode (i.e.accumulating signal over many nominally identical cycles of a process ofinterest), depending on the reproducibility of the sample system beingmeasured.

Referring again to FIG. 8, temporal compressive sensing may beimplemented in an optical system as illustrated for one non-limitingexample. A network of electrically-controllable optical switchesdetermines what fraction of the signal from each time slice reaches eachdetector. This same approach encompasses a variety of differentembodiments, e.g., using free-space optics, fiber optics, or acombination of both; operating in imaging mode, spectroscopy mode, orboth (spectral imaging); using electro-optical and/or acousto-opticalmodulators; using analog or binary modulators (if binary, their speedshould be sufficient to allow many transitions per detector acquisitionperiod); using detectors such as CCD arrays, CMOS arrays, photodiodearrays, or individual high-speed, high-sensitivity detectors such asphotomultiplier tubes; wherein the network topology and the number ofswitches and detectors may vary.

In some embodiment, recombination of signal paths would enableinterferometric operation, particularly if electrically controllablephase shifters are included. This would allow some elements of themeasurement matrix to be negative, thereby providing an advantage insignal-to-noise-ratio-limited operation. It may also enable uniqueholographic temporal reconstruction techniques.

In some embodiment, electronic control systems may switch modulators andtrigger detectors in either a predetermined sequence or an adaptivesequence (i.e. a sequence that can be modified during the acquisition onthe basis of data acquired at any given time). Detectors need not all beoperating at the same frequency.

As with the TEM post-sample-deflector implementation describedelsewhere, the objective is to acquire data from multiple time sliceswithin a single data acquisition period of the detector. If the datastream is highly compressible, the number of time slices reconstructedmay greatly exceed the number of detectors in the system.

Example 4 Stroboscopic Ultrafast TEM

As yet another illustrative (prophetic) example, consider astroboscopic, ultrafast TEM incorporating a picosecond-resolutionarbitrary-waveform laser system as illustrated in FIG. 7. Currently,stroboscopic ultrafast TEM uses a picosecond-scale (orsub-picosecond-scale or femtosecond-scale) electron pulse as a sampleprobe, with one such probe pulse occurring for each cycle of some highlyrepeatable sample process. A time-resolved measurement is performed byaccumulating data from millions of such sample process cycles, shiftingthe time of the probe pulse relative to the phase of the cyclic sampleprocess, and repeating for each time slice to be measured. Measuringhundreds of such time slices can therefore require making measurementsover many billions of cycles of the sample process to be studied, whichmay take many hours. This places extremely high demands on both therepeatability of the sample process and the stability of both the sampleand measurement system. If, instead, each measurement captures data froman arbitrary superposition of time slices, and if we perform multiplemeasurements using such superpositions of time slices, then we have ineffect implemented a temporal compressive sensing system based onequation (3). Such a system could be realized by replacing theshort-pulse laser driving the TEM's cathode with an arbitrary waveformgenerator (AWG) laser system (similar to that described in U.S. Pat. No.9,165,743 but operating on a different time scale), designed so as to beable to produce any specified temporal pattern of light intensity over,for example, a 200 picosecond timespan, with 1 picosecond or betterresolution in the specification of the waveform. This will reduceexperimental data acquisition time through two distinct effects. First,the amount of signal measured per cycle will be greatly increased. Thisis because the amount of current (or electrons per unit time) that canbe used in such a system is limited by space-charge effects (i.e., thefact that electrons repel each other, thus causing the pulse to spreadout in both space and time as it moves from the electron gun to thesample). The proposed arbitrary-waveform laser system would allow thiscurrent limit to be achieved not just for a single ˜1 picosecond timeslice per cycle, but for multiple such time slices. According to CStheory, the optimal data sampling throughput typically occurs at a dutycycle of ˜50%, so in our example of 200 time slices (per 200 picosecondtimespan), ˜100 of the time slices would be filled with electron pulseswhile the rest would be empty. Thus the number of electrons per cyclewould be roughly 100 times more, in this example, for thearbitrary-waveform system than for the single-pulse system, with nocompromise in beam quality or temporal resolution. This means that ˜100times fewer measurement cycles will be required to reach acceptablesignal-to-noise ratios for a given measurement. Second, the number ofsuch measurements should also decrease, because of the inherent natureof compressive sensing such that the number, M, of measurements neededto reconstruct N time slices should be much less than N. Typically theratio M/N is on the order of 0.1, though this varies greatly fromapplication to application. If this ratio holds for the ultrafast TEMapplication, then not only should each of the M acquisitions take 100times less total acquisition time than it would in asingle-pulse-per-cycle system, but the required number of suchacquisitions should be reduced by a factor of ˜10, for an overallreduction in data acquisition time by a factor of about 1,000. Data setscurrently requiring many hours of acquisition time could be acquired inminutes, even including the overhead needed for changing the state ofthe laser system. This represents a dramatic improvement in theperformance of these systems.

Referring again to FIG. 7, a stroboscopic time-resolved TEM using anarbitrary-waveform laser (e.g., with sub-picosecond-scale modulation andsub-nanosecond-scale pulse duration, or with nanosecond-scale modulationand microsecond-scale pulse duration) to modulate the current from aphotoelectron source may be used to implement the compressive sensingmethods of the present disclosure. A second laser beam strikes thesample and initiates the process of interest. Synchronized electrical,micromechanical, or other methods of driving the sample are alsopossible, especially for nanosecond-scale measurements wheretiming-jitter requirements are easily met. The measurement of arepeatable process in the sample is repeated multiple times withdifferent temporal modulation patterns. The mathematical reconstructiontechniques of compressive sensing can then reconstruct the entiresequence of events, with the number of time slices greatly exceeding thenumber of distinct temporal modulation patterns. The time-averaged beamcurrent should greatly exceed that typically used in conventionalultrafast TEM systems, because in the conventional systems the number ofelectrons per pulse is strictly limited by space charge effects and thenecessity to keep the pulse duration at the sample as short as possible.Combining these advantages, the total acquisition time for an experimentcan potentially be reduced by a factor of 1000 or more relative toconventional ultrafast TEM. This dramatically improves one of the mostserious difficulties with conventional ultrafast TEM, namely theextremely long acquisition times and required stability of the sampleunder many millions of measurement cycles.

In other embodiments, alternative beam current modulation techniques,e.g., electrostatic modulation through rapid variation of an electrodesuch as an extractor electrode positioned inside the electron gun, orhigh-speed beam blanking at another location in the column, wouldproduce functionally the same result. The essential point is that thebeam current reaching the detector can be modulated on the time scale ofthe desired time slices.

Example 5 TEM System with High-Speed, Direct Detection Camera

As yet another illustrative (prophetic) example, consider a TEM systemincorporating a high-speed, direct-detection camera, for example, a CMOSframing camera (e.g., a camera that can store multiple images on-chipthrough the use of multiple capacitive bins at each pixel and anelectronic switching system that determines which set of bins isaccumulating signal at any given time) with direct-electron-detectioncapabilities, thereby allowing it to be used for high-speed TEMapplications. With appropriate chip-level electronics design, such adetector could implement the approach described by equation (3) and,with more complexity, even those described by equations (4) or (5)directly. This framing camera approach could also be used for x-raydetection and optical cameras.

All of the illustrative embodiments described above include a commonfeature that is distinct from previous work, i.e., a high-speedswitching and/or modulation system that determines which detector ordetectors selected from a plurality of detectors, or which region orregions selected from a plurality of regions on a single detector,is/are receiving information at any given time. This allowsimplementation of an arbitrary or semi-arbitrary “measurement matrix” ofcoefficients that describe the amount of signal from each time slicereaching each detector or detector sub-region. The mathematicaltechniques associated with compressive sensing then allow reconstructionof a number of individual time slice datasets for each data acquisitionperiod that significantly exceeds (e.g., by 5× to 10×, or more) thenumber of detectors or detector sub-regions.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

What is claimed is:
 1. A method for temporal compressive sensing,comprising: a) directing radiation having an intensity from a sourcetowards a sample or scene; b) capturing sensor array data for one ormore data acquisition periods, wherein within each of the one or moredata acquisition periods, one or more measurement datasets correspondingto distinct linear combinations of patterns of the radiationtransmitted, reflected, elastically scattered, or inelasticallyscattered by the sample or scene are captured for a series of timeslices; and c) reconstructing a time slice dataset for each of the timeslices of the series within each of the one or more data acquisitionperiods using: i) the one or more measurement datasets captured for eachdata acquisition period; ii) a series of coefficients that describe: 1)a known time-dependence of the intensity of the radiation from thesource that is directed to the sample or scene within the dataacquisition period, wherein the coefficients vary as a function of timeslice but are independent of the spatial position for a given pixelwithin the sensor array; or 2) a known time-dependence for switching theradiation transmitted, reflected, elastically scattered, orinelastically scattered by the sample or scene to different regions ofthe sensor array within the data acquisition period, wherein each regionof the sensor array captures a distinct linear combination of patternsof the radiation transmitted, reflected, elastically scattered, orinelastically scattered by the entire sample or scene, and wherein thecoefficients that define the linear combinations vary as a function oftime slice and region of the sensor array but are independent of thespatial position for a given pixel within a given region of the sensorarray; and iii) an algorithm that calculates the time slice datasetsfrom the one or more measurement datasets captured for each dataacquisition period and the series of coefficients; thereby providing aseries of time slice datasets for each of the one or more dataacquisition periods that has a time resolution exceeding the timeresolution determined by the length of the data acquisition period. 2.The method of claim 1, wherein the sensor array is a two-dimensionalsensor array comprising a charge-coupled device (CCD) sensor, acomplementary metal oxide semiconductor (CMOS) sensor, a CMOS framingcamera, a photodiode array, or any combination thereof.
 3. The method ofclaim 2, wherein the sensor array further comprises a nonlinear opticalmaterial, a fluorescent material, a phosphorescent material, or amicro-channel plate, that converts the radiation into radiation directlydetectable by the sensor array.
 4. The method of claim 1, wherein thealgorithm used to reconstruct the time slice datasets is an optimizationalgorithm that penalizes non-sparse solutions of an underdeterminedsystem of linear equations via the l₁ norm, the total number of non-zerocoefficients, total variation, or beta process priors; an iterativegreedy recovery algorithm; a dictionary learning algorithm; a stochasticBayesian algorithm; a variational Bayesian algorithm; or any combinationthereof.
 5. The method of claim 1, wherein at least or at least about 10time slice datasets are reconstructed from the one or more measurementdatasets captured for each data acquisition period.
 6. The method ofclaim 1, wherein the two-dimensional sensor array operates at aneffective data acquisition and read-out rate of at least or at leastabout 100 frames per second.
 7. The method of claim 1, wherein theradiation comprises electrons, and wherein the sensor array is acharge-coupled device (CCD) sensor, an image-intensified charge-coupleddevice (ICCD) sensor, the detector in an electron energy lossspectrometer (EELS), or any combination thereof.
 8. The method of claim1, wherein the radiation comprises electrons and the sensor array isreplaced by the detector in an energy-dispersive x-ray spectrometer(EDX).
 9. The method of claim 1, wherein the time slice data setscomprise reconstructed frames of transmission electron microscope imagedata, transmission electron microscope diffraction pattern data,transmission electron microscope electron energy loss spectral data,transmission electron microscope energy-dispersive x-ray spectral data,or scanning electron microscope image data.
 10. The method of claim 1,wherein the number of time slice datasets to be reconstructed isadjusted during the calculation of the time slice datasets.
 11. Themethod of claim 1, wherein the number of time slice datasets to bereconstructed is optimized by calculating a range of measurement matrixcoefficients, each with a different number of time slices, prior tocapturing the measurement datasets.
 12. The method of claim 1, furthercomprising modulating in a temporal fashion an experimental parameterother than the radiation intensity such that reconstructing the temporaldependence of the time slice dataset for each of the time slicesprovides information on the dependence of the time slice datasets on theexperimental parameter.
 13. The method of claim 12, wherein theexperimental parameter to be temporally modulated is selected from thegroup consisting of rotational orientation of the sample or scene,linear translation of the sample or scene in one dimension, lineartranslation of the sample or scene in two dimensions, and lineartranslation of the sample or scene in three dimensions, or anycombination thereof.
 14. The method of claim 12, wherein the radiationis focused to a narrow beam and the experimental parameter to betemporally modulated is the position of the beam relative to the sampleor scene.
 15. The method of claim 1, wherein the series of coefficientsdescribe: a) a known spatial-dependence and time-dependence of theintensity of the radiation from the source that is directed towards thesample or scene within the data acquisition period; or b) a knownspatial-dependence of the intensity of the radiation from the source anda known time-dependence for switching the radiation transmitted,reflected, elastically scattered, or inelastically scattered by thesample or scene to different regions of the sensor array within the dataacquisition period.
 16. A system for temporal compressive sensing,comprising: a) a radiation source that provides radiation having anintensity directed towards a sample or scene; b) a sensor array thatdetects the radiation subsequent to transmission, reflection, elasticscattering, or inelastic scattering by the sample or scene; c) amechanism that rapidly modulates the intensity of the radiationgenerated by the radiation source prior to its interaction with thesample or scene, or that rapidly switches the radiation transmitted,reflected, elastically scattered, or inelastically scattered by thesample or scene to different regions of the sensor array, and d) one ormore computer processors that: (i) capture sensor array data for one ormore data acquisition periods, wherein within each data acquisitionperiod, one or more measurement datasets corresponding to distinctlinear combinations of patterns of transmitted, reflected, elasticallyscattered, or inelastically scattered radiation for a series of timeslices are captured; and (ii) reconstruct a time slice dataset for eachtime slice within each of the one or more data acquisition periodsusing: 1) the one or more measurement datasets captured for each dataacquisition period; 2) a series of coefficients that describe: i) aknown time-dependence of the intensity of the radiation generated by theradiation source and directed to the sample or scene within the dataacquisition period, wherein the coefficients vary as a function of timeslice but are independent of the spatial position for a given pixelwithin the sensor array; or ii) a known time-dependence for switchingthe radiation transmitted, reflected, elastically scattered, orinelastically scattered by the sample or scene to different regions ofthe sensor array within the data acquisition period, wherein each regionof the sensor array captures a distinct linear combination of patternsof the radiation transmitted, reflected, elastically scattered, orinelastically scattered by the entire sample or scene, and wherein thecoefficients that define the linear combinations vary as a function oftime slice and region of the sensor array but are independent of thespatial position for a given pixel within a given region of the sensorarray; and 3) an algorithm that calculates the time slice datasets fromthe one or more measurement datasets captured for each data acquisitionperiod and the series of coefficients; thereby generating a series oftime slice datasets for each of the one or more data acquisition periodsthat has a time resolution exceeding the time resolution determined bythe length of the data acquisition period.
 17. The system of claim 16,wherein the radiation source is a laser, a photocathode, an electrongun, or any combination thereof.
 18. The system of claim 16, wherein thesensor array is a two-dimensional sensor array comprising acharge-coupled device (CCD) sensor, a complementary metal oxidesemiconductor (CMOS) sensor, a CMOS framing camera, a photodiode array,or any combination thereof.
 19. The system of claim 18, wherein thesensor array further comprises a nonlinear optical material, afluorescent material, a phosphorescent material, or a micro-channelplate, that converts the signal from the radiation source of claim 1into radiation directly detectable by the sensor array.
 20. The systemof claim 16, wherein the algorithm that reconstructs the time slicedatasets is an optimization algorithm that penalizes non-sparsesolutions of an underdetermined system of linear equations via the l₁norm, the total number of non-zero coefficients, total variation, orbeta process priors, an iterative greedy recovery algorithm, adictionary learning algorithm, a stochastic Bayesian algorithm, avariational Bayesian algorithm, or any combination thereof.
 21. Thesystem of claim 16, wherein at least or at least about 10 time slicedatasets are reconstructed from the one or more measured datasetscaptured for each data acquisition period.
 22. The system of claim 16,wherein the two-dimensional sensor array operates at an effective dataacquisition and read-out rate of at least or at least about 100 framesper second.
 23. The system of claim 16, wherein the radiation compriseselectrons and the sensor array is a charge-coupled device (CCD) sensor,an image-intensified charge-coupled device (ICCD) sensor, the detectorin and electron energy loss spectrometer (EELS), or any combinationthereof.
 24. The system of claim 16, wherein the radiation compriseselectrons and the sensor array is replaced by the detector in anenergy-dispersive x-ray spectrometer (EDX).
 25. The system of claim 16,wherein the time slice data sets comprise reconstructed frames oftransmission electron microscope image data, transmission electronmicroscope diffraction pattern data, transmission electron microscopeelectron energy loss spectral data, transmission electron microscopeenergy-dispersive x-ray spectral data, or scanning electron microscopeimage data.
 26. The system of claim 16, wherein the number of time slicedatasets to be reconstructed is adjusted during the calculation of thetime slice datasets.
 27. The system of claim 16, wherein the number oftime slice datasets to be reconstructed is optimized by calculating arange of measurement matrix coefficients, each with a different numberof time slices, prior to capturing the measurement datasets.
 28. Thesystem of claim 16, wherein the series of coefficients describe: a) aknown spatial-dependence and time-dependence of the intensity of theradiation from the source that is directed towards the sample or scenewithin the data acquisition period; or b) a known spatial-dependence ofthe intensity of the radiation from the source and a knowntime-dependence for switching the radiation transmitted, reflected,elastically scattered, or inelastically scattered by the sample or sceneto different regions of the sensor array within the data acquisitionperiod.